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ABSTRACT 



To evaluate the Navy's training under instructional systems 
development (ISO) procedures the annual, force-level, NATOPS 
examination scores of pilots in operational squadrons who had 
been trained under the recently installed ISD program at VP-31 
were compared with those who had not been trained under ISD. 
Scores from only the open- and closed-book written examina- 
tions, which pertain to normal and emergency procedures and 
aircraft systems knowledge, were used since their content re- 
flected most closely the course objectives of the nontactical 
pilot training provided by VP-31. Pilot rank, designation, 
time-in-squadron, and squadron location were used as controls. 
The type of training made no difference in open-book scores. 

For closed-book scores, ISD training resulted in a marginal 
but consistently negative difference as a main effect and in 
a significant negative difference when interactive with 
squadron location. At one location ISD- trained pilots scored 
significantly lower while there was no difference among 
pilots at another location. The findings are discussed with- 
in the context of the study and with respect to training and 
ISD in general. 
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I. 



INTRODUCTION 



A. BACKGROUND 

In support of its aviation mission, the U.S. Navy operates 
training facilities located throughout the country to prepare 
both officers and enlisted personnel for jobs within the 
aviation community. One such facility is Patrol Squadron 
Thirty One (VP-31), which is the Navy's west coast P-3 Orion 
Replacement Air Group (RAG) . VP-31 is responsible for, 

inter alia , the training of maintenance and flight crew 
personnel destined for operational P-3 squadrons located in 
California and Hawaii. 

Prior to the introduction of modern instructional systems 
design (ISD) methods, VP-31 conducted pilot training utiliz- 
ing traditional methods and media consisting primarily of 
classroom lectures, examinations of comprehension, etc. 

Course objectives tended to be more content oriented — "famil- 
iarize the student with performance charts contained in 
chapters 11 and 12" — rather than behaviorally oriented. 
Inherent in these traditional methods was the problem of 
measuring the effectiveness of training. 

B. INSTRUCTIONAL SYSTEMS DEVELOPMENT (ISD) 

Following World War II and the establishment of systems 
analysis as a scientific discipline, psychologists and 
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educators recognized the possible benefits of applying 
systems technology to the field of instruction. These 
pioneers of modern instructional technology, who came not 
only from academia but also from quasi-governmental agencies 
such as RAND and HUMRRO, saw great potential in the use of 
techniques such as modelling to increase the precision, 
objectivity, and creativity of the judgment they exercised 
in solving the problems of course design [Montemerlo, 1979]. 
Perhaps the most significant contribution to the field of 
instruction by training technologists has been a demanding 
emphasis upon the precise definition of learning objectives 
and goal-oriented performance evaluation. Enabling objec- 
tives, final objectives, and course objectives form a hierarchy 
which serves to, among other things, provide a clear pathway 
for both students and instructors, and to answer the question — 
"precisely what will a student be able to do after success- 
fully completing a course of instruction?" Goal-oriented 
performance evaluation provides a method to measure accurately 
the effectiveness of training programs that are oriented 
toward producing graduates capable of actually doing the 
job. 

Instruction can be divided into five categories represen- 
ting learned capabilities, i.e., verbal information, cog- 
nitive strategies, intellectual skills, motor skills, and 
attitudes [Gagne & Briggs, 1979]. Associated with each 
variety of learned capability are conditions for learning. 
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Traditional instructional methods have always emphasized 
external conditions for learning such as repetition and stim- 
ulous-response reinforcement. Modern theories of learning 
focus also upon the internal conditions for learning such as 
prior learning, recall, and various levels of memory [Gagne 
& Briggs , 1979] . 

Taxonomies such as the five types of learning described 
by Gagne serve to group similar objectives in a course design 
and provide for mutually applicable learning conditions . 
Additionally, grouping facilitates the employment of like 
methods of evaluation. Performance measures of a motor skill 
such as the manipulation of aircraft controls requires a 
student to perform bodily movements involving muscular activity. 
The performance measure appropriate to the learning of verbal 
information requires a student to state the relationship 
between two or more named objects or events either orally or 
in writing. 

Attitudes are complex states of human beings relating to 
things, people, or events which can be described in terms of 
either their cognitive or affective aspects. Measurement of 
attitudes is most logically suited to the affective aspect 
and therefore centers upon observing a person's chosen al- 
ternative course of action from a selection of options. 

Intellectual skills form a hierarchy which starts with 
simple discriminations and concrete concepts. Simple rules, 
which include defined concepts, follow discriminations and 
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lead to problem solving through higher-order rules. Perfor- 
mance measures of intellectual skills range from the basic 
act of distinguishing objects to the formulation and employ- 
ment of complex rules in order to achieve solutions to 
unique problems which are new to the student. 

Cognitive strategies are special forms of intellectual 
skills. These special skills actually represent processes 
by which learners modify their ways of attending, learning, 
remembering, and thinking. The salient difference between 
intellectual skills in general and cognitive strategies is 
the object of the skill. Cognitive strategies focus upon 
the learner's own thought process whereas intellectual skills 
relate to objects and events [Gagne S Briggs, 1979]. The 
performance measure of a cognitive strategy requires the 
novel solution to a problem where familiar solutions are 
not suitable. 

Goal-oriented performance objectives are requisite 
measures of learning success in ISO programs. However, the 
question of — "performance quality" must be considered. 
Objective-referenced examinations employing a criterion- 
referenced interpretation provide the most acceptable pro- 
cedure for a variety of reasons. The validity of an objective- 
referenced test is directly related to its congruence with 
the course objective. Reliability of such a test is derived 
from the degree of consistency of repeated examinations. 

In an objective-referenced test of intellectual skills. 
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motor skills, or verbal information, mastery is a relevant 
concept and error-free performance can be achieved. However 
cognitive strategies and attitudes are not suited to the 
mastery criterion because of their lack of an absolute 
measure [Gagne & Briggs, 1979]. 

Group instruction can be regarded in three separate 
categories which differ in the amount of precision afforded 
to the management of instructional events by the instructor. 
In a two-person tutorial situation, the instructor is pro- 
vided with the most control over instructional events. A 
small group is defined as three to eight people and is 
dependent upon group interaction to provide instructional 
events such as feedback, recall, etc. [Gagne & Briggs, 1979] 
Large groups of more than 15 people are characterized by 
weak control over instructional events by the instructor. 
Accordingly, in a large group, learning is highly dependent 
upon the learner's own strategies of self instruction. To 
counter the inherent weak control of learning in large 
groups, the mastery concept for enabling objectives serves 
to diagnose learning performance early on and facilitate 
attainment of the course objectives in ISO programs. 

Individualized instruction is designed in the same 
fashion that individual lessons are designed for group in- 
struction. Objective hierarchies, learning events, and 
conditions for learning all apply to the design of modules 
suited to individualized instruction. The distinction lies 
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in the delivery system employed for individualized instruction. 
Modules are normally self-paced and the materials do most of 
the teaching. More frequent feedback and progress checks are 
found in programs of individualized instruction. Under an 
ISO system, students are typically provided a series of 
modules v;hich each contain: 

1. A performance objective 

2. A set of materials and learning activities 

3. A method of self-evaluation to determine mastery 

4. A provision for verification of learning to the 
instructor [Gagne & Briggs, 1979] 

As an organization responsible for a myriad of training 
programs ranging in subject matter from remedial reading to 
the conduct of war on a global basis, it was not at all 
surprising that the U.S. military establishment became in- 
terested and involved in the methods of ISD as it gained in 
credibility. With its basis in scientific methods and logical 
appeal, ISD became an attractive alternative to traditional 
training methods which offered the unique advantage of a 
capability to measure the effectiveness of a training program 
in terms of job-related performance. 

Today, ISD is an integral part of the training conducted 
by all four services. ISD has been employed by the U.S. 
military to ensure that; 

1. Courses are designed to teach only those tasks 
which, based upon objective field research and 
analysis of the tasks needed to be performed, 
the graduate will use and which can be taught 
in a formal training course. 
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2. Tests, the requisite for graduation, are accurate 
indicators of the ability to perform the required 
task [Department of Defense, 1980 p. XI-7] . 

The five phase ISD program, — i.e., analyze, design, 
develop, implement, and control — and associated phase steps 
is presented in Figure 1. 

The Navy is committed to the systematic design of its 
training courses [Chief of Naval Education and Training (CNET) 
1975; Scanland, 1974]. The Interservice Procedures for In- 
structional Systems Development (Chief of Naval Education 
and Training) have been summarized by the Navy for its own 
use in a procedures manual entitled Procedures for Instruction 
al Systems Development [CNET, 1978]. AF Pamphlet 50-58 [De- 
partment of the Air Force, July 197 S] describes the Air Force 
ISD procedures and AF Manual 50-2 [Department of the Air 
Force, July 1975] promulgates the following policy regarding 
the utilization of ISD. 

1. Apply ISD to all new instructional systems 

2. Selectively apply ISD to existing instructional 
systems where economically feasible 

The extensive use of ISD by the military has prompted 
periodic review of training programs [Brock, 1977; Logicon 
Incorporated, 1978] . These studies emphasize the importance 
of job-related learning objectives as a prerequisite to 
successful ISD programs. If lists of learning objectives 
are provided for students, it seems natural, however, that 
their efforts in learning will be generally limited to 
those items on the list. Consequently, ISD program design- 
ers must leave no stone unturned in an exhaustive search 
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Figure 1. The Instructional Systems Development Process 
[CNAVEDTRA 11 O] 
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to identify all relevant items in a task analysis. Under 
ISO, it is naive to assume that the learning of important but 
easily overlooked items, such as a desired attitude or a 
knowledge of the relationship of components to a total system, 
will take place in the absence of specifically related 
learning objectives. 

Montemerlo [1979] cautions designers of instructional 
systems against a potential problem associated with the 
atomization approach of ISO manuals. He alleges that students 
inundated with a torrent of modularized learning objectives 
may miss an overall goal. Atomistic descriptions tend to 
feed the learner a large number of nev; terms, definitions, 
and relationships faster than he can assimilate them [Monte- 
merlo, 1979]. The spiral approach to learning, which is 
the basis for most apprenticeship programs, focuses upon 
progressively more complex hands-on practice interspersed 
with limited book-learning. It is because of channel capacity, 
which is the amount of information which a person can process 
before becoming confused, that Montemerlo [1979] asserts 
spiral learning works better than atomistic descriptions. 

Training is an investment in human capital and represents 
an organizational asset. Associated with a training invest- 
ment in human capital is an allocation of resources, such 
as time and money. The ISD method of instruction is extremely 
costly in terms of time required to develop and implement a 
program. Lexicon Incorporated, [1978] have tentatively 
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suggested an excess of five years is required, under ideal 
conditions, for the conversion of typical flying programs. 
However, ISD programs, with self-instruction, offer the 
potential of reducing the manhours spent by instructors in 
teaching. In order to determine accurately the total cost 
of an ISD application, the extensive costs related to 
development of the program must not be overlooked as these 
front-end costs are significant. 

The evaluation of a learning program is typically cen- 
tered upon three questions: 

1. To what extent have the stated objectives of 
instruction been met? 

2. In what ways and to what degree is it better than 
the unit (of instruction) it will supplant? 

3. What additional, possibly unanticipated, effects 
has it had and to what extent are these better or 
worse than the supplanted unit? [Gagne & Briggs, 

1979] 

Formative evaluation is normally conducted during the 
development of a new system and serves to enhance the final 
product through modification or revision. By formative 
evaluation, quality in the design of a program can be ensured. 
Summative evaluation, on the other hand, is performed upon 
an existing course of instruction and normally serves to 
compare a new system to an "old one." During a summative 
evaluation, variables such as student aptitude for learning, 
the process of instruction, and learning support items must 
be controlled if a truly meaningful and unbiased evaluation 
is to result. 
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The training of Naval Aviators for duty as pilots and 
copilots in P-3 aircraft involves primarily the learning 
taxonomies of motor skills , verbal information, and intel- 
lectual skills. However, positive attitudes towards things 
such as safety and furthering aviator skills should also be 
considered germane to a total training program. There is 
an emerging acceptance of ISD as a total systems approach 
rather than a series of steps to be accomplished on some 
segment or portion of flying training. This acceptance is 
primarily responsible for the fact that ISD teams are 
beginning to return to the task analysis in order to 
strengthen it by means of a systematic, rather than a 
cursory or short-cut approach [Lexicon Incorporated, 1978, 
p. 58]. 

The purpose of this study was to examine the effective- 
ness of the application of ISD to VP-31 pilot training as 
measured by scores achieved upon annually administered, 
force-level, NATOPS examinations. Another study currently 
underway [Fadness, 1980], has been designed to examine the 
application of ISD to the training of P-3 acoustic sensor 
operators at VP-31. 
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II. METHOD 



A. APPROACH 

In order to evaluate the effectiveness of pilot training 
at VP-31 under a new ISD program, a sample of both ISD and 
non-ISD trained graduates was required. Under ideal circum- 
stances, an experimental group (ISD) and an identical control 
group (non-ISD) would be trained in parallel under conditions 
as identical as possible. Following graduation, the training 
received by both groups would be evaluated using identical, 
job-related performance measures. Further, retention of 
training could be measured at a later time. 

Under the constraints that this study experienced, the 
experimental and control groups had to be reconstructed from 
individuals already in operational squadrons located at two 
west-coast P-3 bases who had received training under one 
condition or the other. Since the training received by these 
individuals had occurred over a considerable period of time, 
the individuals varied extensively in both pre- and post- 
training experiences. Accordingly, demographic data v;ere 
collected to control for these differences statistically. 
Scores achieved upon annually administered force level 
standardization examinations were selected as the dependent 
variable because of the relatively constant examination 
content over the time period examined. 
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B. DESCRIPTION OF ANNUALLY ADMINISTERED STANDARDIZATION 

EXAMINATIONS 

Standardization of normal and emergency operating proce- 
dures for crewmembers of Naval aircraft is provided by a 
program known as Naval Air Training and Operating Procedures 
Standardization (NATOPS) . Included within the NATOPS program 
for P-3 Orion pilots is a squadron-level, initial certifica- 
tion to serve in the capacity of pi lot- in-command and subsequent 
annual recertifications. Each certification process consists 
of two written examinations (one open book and one closed 
book) and an inflight check of piloting skills. Each phase 
of this examination process is designed to evaluate stated 
skill areas. Open-book examinations contain 24 multiple- 
choice questions relating primarily to performance charts 
located within the NATOPS manual. Pilots are required to 
compute various performance values from charts and select 
the correct answer to a written question from four choices. 
Closed-book examinations contain 40 multiple choice questions 
relating to normal and emergency procedures and aircraft 
systems. Pilots are required to respond to a written question 
by selecting one of four possible answers. Flight checks, 
which last approximately three hours, are designed to evaluate 
both knowledge of procedures and systems (verbal answers to 
verbal questions), and piloting skills. The content of written 
examinations must follow a distribution of subject matter 
--number of questions per subject — specified by the force- 
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level evaluation team. To assist in examination construction, 
the force-level NATOPS evaluation team maintains a sizeable 
bank of sample questions for each subject area. Objective 
performance criteria for flight checks is provided within the 
NATOPS program to aid in standardization. 

In addition to squadron level NATOPS examinations, force 
level NATOPS examinations are administered to each operational 
squadron on an annual basis. Force-level examinations consist 
of two written examinations (one open book and one closed 
book) for all crewmembers and flight checks for only three 
selected personnel per crew position. Force- level evaluations 
are intended to be similar in content to squadron-level 
evaluations. Crewmembers certified by the NATOPS force-level 
evaluator are awarded squadron NATOPS instructor status and 
are authorized to conduct squadron-level NATOPS check flights 
for their crew position. The results of force-level NATOPS 
unit evaluations for west-coast (California and Hawaii) P-3 
squadrons are maintained by the COMNAVAIRPAC P-3 NATOPS 
evaluation team. Force level NATOPS written examination 
scores achieved by P-3 pilots between 1978 and 1980 were 
made available for this study by the Commanding Officer, 

Patrol Squadron Thirty One, and the COMNAVAIRPAC NATOPS 
pilot evaluator. It was agreed that the names of individual 
pilots and their respective squadrons would remain anonymous 
throughout the analysis. 
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C. SAMPLE DESCRIPTION 

Between January 1978 and May 1980, 31 west coast, opera- 
tional P-3 squadrons received force level, NATOPS, unit 
evaluations. In the Spring of 1979, graduates of VP-31 's new 
ISD training program began reporting to west-coast P-3 squadrons 
and participating in scheduled, force-level, NATOPS examina- 
tions. Three hundred eighty-two pilots from 10 squadrons 
located in California and Hawaii were selected as the initial 
sample. Composition of the initial sample was based upon two 
primary factors: 

1. Obtaining a maximum number of ISD-trained pilots 

2. The deployment status of squadrons which determined 
their ability to provide demographic data on sample 
pilots 

Demographic data regarding the sample pilots were provided by 
squadrons through crew lists applicable to the date of the 
unit evaluation. 

During initial analysis it was not surprising to discover 
that because of the January 1979 implementation date of ISD 
at VP-31, sample pilots trained under the ISD condition had 
generally been onboard their assigned operational squadrons 
only a few months. Accordingly, the sample was reduced to 
those pilots with 12 or fewer months experience onboard their 
assigned squadrons. This reduction provided for a total of 
123 total pilots of which 61 were trained under the ISD 
program (Tables 1 and 2) . 



23 



Table 1 



Sample of Pilots Used in the Study by Rank, Months 
in the Squadron, Type of Training and Location* 



RANK 




MONTHS 




TOTAL 




1 

1 — 1 


5-8 


9-12 





Location A 



0-5 


2 


( 


1) 


0 


( 


1) 


1 


( 


1) 


3 


( 


3) 


0-4 


2 


( 


5) 


4 


( 


2) 


2 


( 


2) 


8 


( 


9) 


0-3 


1 


( 


1) 


2 


( 


1) 


2 


( 


2) 


5 


( 


4) 


0-2 


6 


(12) 


8 


( 


6) 


14 


( 


2) 


28 


C 


20) 


0-1 


1 


( 


2) 


2 


( 


0) 


0 


( 


0) 


3 


( 


2) 


Sub-total 


12 


(21) 


16 


(10) 


19 


( 


7) 


47 


C 


38) 



Location B 



0-5 


0 


( 


0) 


1 


( 


0) 


0 


( 


1) 


1 


( 1) 


0-4 


0 


( 


3) 


1 


( 


3) 


0 


( 


0) 


1 


( 6) 


0-3 


0 


( 


1) 


1 


( 


0) 


1 


( 


0) 


2 


( 1) 


0-2 


1 


( 


7) 


5 


( 


7) 


5 


( 


0) 


11 


(14) 


0-1 


0 


( 


1) 


0 


( 


0) 


0 


( 


0) 


0 


( 1) 


Sub-total 


1 


(12) 


8 


(10) 


6 


( 


1) 


15 


(23) 


Total 


13 


(33) 


24 


(20) 


25 


( 


8) 


62 


(61) 



* Numbers in parenthesis represent those who were trained 
under the ISD condition 
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Table 2, 



Sample of Pilots Used in the Study by Rank, 
Designation, Type of Training, and Location* 



RANK 




DESIGNATION 




TOTAL 




NOP 


PP3P PP2P 


PPC 





Location A 



0-5 


1 


( 


0) 


0 


( 


0) 


0 


(1) 


2 


(2) 


3 


( 


3) 


0-4 


3 


( 


3) 


0 


( 


0) 


1 


(2) 


4 


(4) 


8 


( 


9) 


0-3 


3 


( 


1) 


2 


( 


3) 


0 


(0) 


0 


(0) 


5 


( 


4) 


0-2 


8 


(14) 


18 


( 


6) 


2 


(0) 


0 


(0) 


28 


(20) 


0-1 


2 


( 


1) 


1 


( 


1) 


0 


(0) 


0 


(0) 


3 


( 


2) 


Sub-total 


17 


(19) 


21 


(10) 


3 


(3) 


6 


(6) 


47 


(38) 



Location B 



0-5 


0 


( 0) 


0 


( 


0) 


0 


(0) 


1 


(1) 


1 


( 1) 


0-4 


1 


( 5) 


0 


( 


0) 


0 


(0) 


0 


(1) 


1 


( 6) 


0-3 


0 


( 1) 


2 


( 


0) 


0 


(0) 


0 


(0) 


2 


( 1) 


0-2 


5 


(10) 


6 


( 


4) 


0 


(0) 


0 


(0) 


11 


(14) 


0-1 


0 


( 1) 


0 


( 


0) 


0 


(0) 


0 


(0) 


0 


( 1) 


Sub- total 


6 


(17) 


8 


( 


4) 


0 


(0) 


1 


(2) 


15 


(23) 


Total 


23 


(36) 


29 


(14) 


3 


(3) 


7 


(8) 


62 


(61) 



Numbers in parenthesis represent those who were trained under 
the ISD Condition 
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D. DEPENDENT VARIABLES 

Force-level NATOPS written examination scores achieved 
by the 123 pilots during unit evaluations were selected as 
the dependent variable on which to evaluate phase A and B 
pilot training conducted at VP-31. The suitability of force- 
level written examinations as a training effectiveness 
measure is established by the close agreement between the 
course objectives of phase A and B pilot training and the 
subject matter specified for testing in the P-3 NATOPS 
program. During phase A and B, pilot training subject matter 
consists of normal and emergency procedures and the operation 
of aircraft systems. It is during phase C and D, which have 
been purposely excluded from this study, that training in the 
various aspects of ASW and tactics is conducted. 

E. INDEPENDENT VARIABLES 

Various demographic variables in addition to the type of 
training received were considered, and the following five were 
ultimately selected (See Table 3) . 

1 . Instructional Systems Development (ISP) 

The Instructional Systems Development variable iden- 
tifies the type of training that a pilot within the sample 
received while attending VP-31. Those pilots who were trained 
under the new system implemented in January of 1979 were 
coded 1, and those trained under the old system v/ere coded 
0 . 
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Table 3 



Definition of Variables 



NAME VALUE LEGEND 



Pilot Rank 1 

(RANK) 2 

3 

4 

5 

Designation 1 

(DESIG) 2 

3 

4 

Months 1 

(DMONTHS) 2 

3 

Training 0 

(ISD) 1 

Squadron 1 

(DSQD) 2 



Ensign 

Lieutenant Junior grade 
Lieutenant 

Lieutenant Commander 
Commander 

Patrol Place Commander (PPC) 

Patrol Plane Second Pilot (PP2P) 

Patrol Plane Third Pilot (PP3P) 

Non-Designated Pilot (NOP) 

1-4 Months Onboard Squadron at Time 
of Test 

5-8 Months Onboard Squadron at Time 
of Test 

9-12 Months Onboard Squadron at Time 
of Test 

Non-ISD-Trained Pilot 
ISD-Trained Pilot 

All Squadrons Located at Location A 
All Squadrons Located at Location B 
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2. Pilot Rank 



The rank of all sample pilots at the time they took 
the annual NATOPS examination was coded from 1 to 5 represent- 
ing the prescribed military structure of 0-1 through 0-5. 

3 . Designation 

Pilots in the P-3 community are grouped into four 
general categories which are determined by their qualification 
as pilots. Nonrated pilots (NOP) are those pilots who have 
not yet qualified. Patrol Plane Third Pilots (PP3P) have 
successfully completed the first stage of the P-3 pilot 
Personal Qualifications Standards (PQS) training but have 
not yet completed a NATOPS check. Patrol Plane Second Pilots 
(PP2P) have successfully completed the second stage of PQS 
training and a pilot NATOPS check. A PP2P is considered 
"safe for solo" and may serve as pilot-in-command during a 
nontactical mission. Patrol Plane Commanders (PPC) have 
successfully completed the required PQS and the training 
directed by the Commander, Patrol Wings Pacific (CPWP) . 

Patrol Plane Commanders are authorized to serve as pilot-in- 
command of tactical missions. 

4 . Months 

The months variable represents the number of months 
that a pilot in the sample was onboard his squadron at the 
time of the force-level NATOPS examination. Although this 
variable ranged from 1 to 36, the sample, as stated, was 
restricted to those pilots with from 1 to 12 months onboard 
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a squadron when the examination was administered. This 
restriction reduced the sample size to 123 but provided for 
a more representative distribution of the ISD variable due 
to the newness of ISD at VP-31. To facilitate analysis, 
aggregation of the months variable resulted in three groups: 
1-4 months, 5-8 months, and 9-12 months. 

5 . Squadron 

The squadron variable served to distinguish the vari- 
ous squadrons to which west coast P-3 pilots are assigned 
subsequent to VP-31 training. Aggregation of the squadron 
variable resulted in two groups representing the two bases 
at which squadrons are located. 

In the case of all variables, aggregation was identi- 
fied by a preceding letter D. 

F. ANALYSIS OF THE DATA 

Analysis of the data was conducted at three levels to 
investigate all potential aspects of the problem. Initial 
analysis focused upon the distribution of open- and closed- 
book scores by demographic variables. Additionally, tests of 
independence between the distribution of the subjects on the 
ISD variable and each of the demographic variables were com- 
puted and examined using chi-square. Utilizing further 
aggregated variables (see Results section) , a2x2x2x3 
factorial analysis of variance of open- and closed-book 
examination scores was conducted. This investigation utilized 
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a Statistical Package for the Social Sciences [Nie, Hull, 
Jenkins, Stienbrenner & Bent, 1978] computer program and was 
designed to reveal the significance of four demographic 
variables (ISD, DSQD, DRANK, and DMONTHS) in accounting for 
the heterogeneity of open- and closed-book examination scores. 
Due to missing DDESIG data points in the sample (Table 2) , 
the DDESIG variable was excluded from the factorial analysis 
of variance to permit the computation of two-way interactions. 
Further, in view of the high degree of dependence between 
the DDESIG and DRANK variables (Chi-square test of indepen- 
dence significant at <.001), exclusion of the DDESIG variable 
would not detract from the analysis. Three-way or higher 
interactions were pooled with the error sum of squares. Ag- 
gregated variables were again used for multivariate analysis 
of the variance among open- and closed-book examination 
scores considered simultaneously. This analysis was executed 
utilizing the Statistical Analysis System [SAS Institute, 

1979] computer program and examined the effects of all five 
demographic variables in accounting for the variance of com- 
bined scores. 

In addition to the statistical analyses described, 
operational factors associated with the conduct of training 
at VP-31 were examined. Fiscal year averages of salient 
measures were compared for FY77, FY78 and FY79. Factors 
such as staff size, instructor complement, student load, time 
spent in ground school during the familiarization phase, and 
total syllabus length were considered germane to the study. 
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III. RESULTS 



A. SUMMARY RESULTS 

Based upon the initially defined independent variables, 
means and standard deviations for open- and closed-book ex- 
aminations were computed by category of training received at 
VP-31 (See Table 4) . 

B. AGGREGATED VARIABLES 

Further analysis of the problem was enhanced by aggrega- 
tion of the rank and designation variables to permit the use 
of multivariate procedures. The five-category rank variable 
was aggregated into two groups: 0-1 through 0-3 and 0-4 
through 0-5. This distribution provided for not only a 
grouping of junior and senior officers, but additionally, it 
provided an identification of first- and second-tour pilots. 
The four level designation variable was combined to achieve 
two levels of designation: NOP and PP3P, and PP2P and PPG. 
This division provided a means to identify pilots who had 
completed a NATOPS certification check. The squadron vari- 
able and months variable were already aggregated (See Table 
3) . Means and standard deviations for the newly aggregated 
rank and designation variables are presented in Table 5. 

Tables 6 and 7 present a contingency analysis of the 
training variable with each of the aggregated demographic 
variables to check for independence of the training variable 
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Table 4 



Means and Standard Deviations of Annual 
NATOPS Open and Closed Book Examinations 
by Category of Training Received* 



DEMOGRAPHIC VALUE TEST 

OFlN CLOSED 







ISD 


NON- 

ISD 


ALL 


ISD 


NON- 

ISD 


ALL 


Pilot Rank 


1 


3.55 

.48 


3.44 

.34 


3.50 

.38 


3.33 
. 59 


3.60 

.17 


3.47 

.41 


(RANK) 


2 


3.69 

.31 


3.73 

.25 


3.71 

.28 


3.42 

.39 


3.62 

.27 


3.53 

.35 




3 


3.60 

.32 


3.51 

.53 


3.55 

.44 


3.32 

.39 


3.54 

.30 


3.45 

.34 




4 


3.85 

.18 


3.89 

.17 


3.87 

.17 


3.74 

.24 


3.77 

.30 


3.75 

.26 




5 


4.00 

.00 


3.79 

.16 


3.90 

.15 


3.68 

.17 


3.78 

.26 


3.73 

.21 


Designation 


1 


3.94 

.09 


3.86 

.15 


3.90 

.12 


3.73 

.21 


3.81 

.20 


3.77 

.21 


(DESIG) 


2 


3.94 

.10 


3.94 

.10 


3.94 

.08 


3.83 

.15 


3.70 

.26 


3.77 

.21 




3 


3.74 

.28 


3.75 

.27 


3.75 

.27 


3.46 

.33 


3.59 

.30 


3.55 

.31 




4 


3.67 

.32 


3.60 

.34 


3.64 

.33 


3.44 

.42 


3.64 

.25 


3.52 

.37 


Months 


1 


3.66 

.32 


3.58 

.39 


3.63 

.34 


3.43 

.44 


3.56 

.31 


3.47 

.41 


(DMONTHS) 


2 


3.83 

.25 


3.72 

.28 


3.77 

.27 


3.58 

.28 


3.63 

.29 


3.61 

.30 




3 


3.81 

.23 


3.79 

.24 


3.79 

.23 


3.63 

.29 


3.69 

.25 


3.67 
. 26 
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Table 4 (continued) 



Squadron 


1 


3.69 


3.68 


3.69 


3.48 


3.67 


3.59 






.34 


.31 


.32 


.43 


. 27 


.36 


(DSQD) 


2 


3.80 


3.83 


3.81 


3.53 


3.54 


3.54 






.20 


.22 


.20 


.30 


.26 


.29 


TOTAL 




3.73 


3.72 


3.73 


3.50 


3.64 


3.57 






.30 


.30 


.30 


.38 


.28 


.34 



* 

Means above and standard deviations below in each set of 
data 
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Means above and standard deviation below in each set of data 



Table 6 



Contingency Analysis of the Distribution 
of ISD and Non-ISD Trained Pilots 
by Demographic Variables 



DEMOGRAPHIC 

VARIABLES 


VALUES 


ISD 


NON- 

ISD 


CHI 

SQUARE 


SIGNIFIANCE 
OF CHI SQUARE 


DRANK 


1 


42 


49 


1.17 


.30 




2 


19 


13 






DSQD 


1 


38 


47 


2.03 


.15 




2 


23 


15 






DDESIG 


1 


11 


10 


.00 


.97 




2 


50 


52 






DMONTHS 


1 


33 


13 


17.81 


.00 




2 


20 


24 








3 


8 


25 
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Table 7 



Contingency Analysis of the Distribution 
of ISD and Non-ISD Trained Pilots 
by Aggregated Months and Location 



NON- CHI SIGNIFICANCE 

LOCATION DMONTHS ISD ISD SQUARE OF CHI SQUARE 



A 



1 21 12 



8.52 



.01 



2 10 16 



3 7 19 



B 



1 12 



1 11.95 



. 00 



2 10 8 



3 16 
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distribution. There was a significant degree of dependence 
in the distribution of the ISD variable by DMONTHS at both 
squadron locations examined. This result was not surprising 
since pilots trained under the new ISD system were more 
likely to have fewer months onboard their squadron than non- 
ISD trained pilots at the time of the unit NATOPS evaluations. 
The adjusted chi-square computed for the DMONTHS-ISD matrix 
was 17.8 and significant at the .0001 level. 

C. FACTORIAL ANALYSIS OF VARIANCE OF OPEN BOOK SCORES 
(TABLE 8) 

The DRANK, DSQD, and DMONTHS variables were found to be 
significant as main effects in accounting for the observed 
heterogeneity among sample open-book NATOPS examination 
scores. Additionally, the interaction between DSQD and 
DRANK variables was determined to be significant. Mean 
open-book examination scores for both values of the DRANK 
variable were demonstrated to be nearly identical at loca- 
tion B and quite different at location A. Accordingly, 
although the DRANK and DSQD variables were significant when 
considered separately, the predictive power of the DRANK 
variable was enhanced by knowing the location variable 
(See Table 9 and Figure 2) . It is interesting to note that 
in spite of the significant degree of dependence in the 
distribution of ISD by DMONTHS, the tv/o-way interaction 
between these variables in the factorial analysis was not 
significant. Further, ISD, per se , was not identified as 
a significant effect in this analysis. 
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Table 8 



Factorial Analysis of Variance of Open-Book 
NATOPS Scores by Demographic Variables 



DEMOGRAPHIC VARIABLE 


D/F* 


F VALUE 


SIGNIFICANCE 
OF F 


ISD 




1 


.577 


.45 


DMONTHS 




2 


5.075 


.01 


DRANK 




1 


14.151 


.00 


DSQD 




1 


6.002 


.02 


Two Way 


Interactions 








ISD 


DMONTHS 


2 


.072 


.93 


ISD 


DSQD 


1 


.514 


.48 


ISD 


DRANK 


1 


.041 


.84 


DMONTHS 


DSQD 


2 


.992 


.37 


DMONTHS 


DRANK 


2 


.279 


.76 


DSQD 


DRANK 


1 


6.233 


.01 



*Total D/F = 122 
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Table 9 



Mean Open-Book NATOPS Scores for Location (DSQD) 

by Rank (DRANK) 



Value of DRANK 



1 

2 



Location 



A 


B 


3.61 


3.82 


3.9 


3.81 



39 



SCORE 




Location A 
Location B 



DRANK 



Figure 2, Interactive Effects of Rank and Location 
on Open-Book NATOPS Examination Scores 
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D. FACTORIAL ANALYSIS OF THE VARIANCE OF CLOSED-BOOK 

SCORES (TABLE 10) 

The factorial analysis of variance of closed-book scores 
identified the DMONTHS and DRANK variables as significant 
main effects. Although not significant at the .05 level, 
the ISD variable was significant at the .08 level. Further, 
the two-way interactions of ISD with DSQD and DSQD with 
DRANK were significant in accounting for score variance. 
Details of these interactions are shown in Table 11 and 
Figure 3. Pilots stationed at location B were observed to 
achieve nearly idential mean scores on the NATOPS closed- 
book examinations regardless of the type of training received 
while at VP-31 or their current rank. However, the mean 
scores achieved by pilots stationed at location A varied 
considerably by the type of training received while at VP-31, • 
the non-ISD- trained group had a higher mean score than the 
ISD-trained group. The difference in these two means, as 
measured with a t test, was significant at the .01 level. 
Additionally, although the DRANK variable made no difference 
in the mean closed-book score at location B, it did account 
for a meaningful difference in mean scores at location A in 
that the pilots with higher rank, and presumably more exper- 
ience, made better scores. 

E. MULTIVARIATE ANALYSIS OF VARIANCE OF OPEN- AND CLOSED- 

BOOK NATOPS SCORES (TABLE 12) 

A multivariate analysis of open- and closed-book NATOPS 
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Table 10 



Factorial Analysis of Variance of Closed-Book 
NATOPS Scores by Demographic Variables 



DEMOGRAPHIC VARIABLE 


D/F* 


F VALUE 


SIGNIFICANCE 
OF F 


ISD 




1 


3.051 


.08 


DMONTHS 




2 


3.032 


.05 


DSQD 




1 


.177 


.67 


DRANK 




1 


16.271 


.00 


Two Way 


Interactions 








ISD 


DMONTHS 


2 


.051 


.95 


ISD 


DSQD 


1 


4.618 


.03 


ISD 


DRANK 


1 


1.806 


.18 


DMONTHS 


DSQD 


2 


.391 


.68 


DMONTHS 


DRANK 


2 


.598 


.55 


DSQD 


DRANK 


1 


7.842 


.01 



*Total D/F =122 
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Table 11 



Mean Closed-Book NATOPS Scores for Location 
(DSQD) by Training (ISD) and Rank (DRANK) 



VARIABLE 


VALUE 




LOCATION 








A 




B 


ISD 


0 


3.67 




3.54 




1 


3.48 




3.53 


DRANK 


1 


3.50 




3.54 




2 


3.83 




3.52 
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Score 




Score 




ISD 



Figure 3. Interactive Effects of Location (DSQD) 
with (a) DRANK and (b) ISD on Closed- 
Book NATOPS Examination Scores 



44 



Table 12 



Multivariate Analysis of Variance of Annual NATOPS 
Open- and Closed-Book Examination Scores 



HYPOTHESIS OF NO OVERALL 

EFFECT BY: F* SIGNIFICANCE OF F 



ISD 


2.05 


.13 


DRANK 


4.67 


.01 


DMONTHS 


2.53 


- .04 


DDESIGN 


0.63 


.54 


DSQD 


3.90 


.02 



*F Determined by Wilk's criterion where; 



F= 



1 - VT 



X 



(NE+Q-P-1) 

P 



with 2P and 2 (NE+Q-P-1) d/f 



DET(E) 

^ DET (H+E) 

H = sum of squares and cross product matrix for the null 
hypothesis 

E = error sum of squares and cross product matrix 
P = number of dependent variables 
Q = degree of freedom of null hypothesis 
NE = D/F of error 
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examination scores, considered simultaneously, identified 
the DRANK, DMONTHS, and DSQD variables as significant in 
accounting for score heterogeneity. Because of the previous- 
ly discussed sensitivity of open- and closed-book NATOPS 
scores to the experience variables (DRANK and DMONTHS) and 
the squadron location variable (DSQD) , the multivariate 
analysis was performed to test the power of these effects 
on the combined scores. Although the correlation between 
open- and closed-book scores was computed to be only .33, 
the effects of squadron location and both experience vari- 
ables were again statistically significant in the multi- 
variate analysis. All interactive effects were excluded 
from this test because of a singular matrix caused by missing 
data points for the DDESIG variable which prohibited the 
computation of two-way and higher interactions. As in the 
factorial analyses, the ISD variable was not statistically 
significant. 

F. OPERATING DATA COMPARISON (TABLE 13) 

The most noticeable changes in operational factors over 
the time frame examined (FY77 through FY79) have occurred 
in the following four areas: 

1. Student/instructor ratio 

2. Second tour pilot load 

3. Ground school time 

4. Enlisted staff size 
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Table 13 



A Comparison of Operating Data for VP-31 Training 



FACTOR 


TYPE OF FACTOR 


FY77 


FY78 


FY79 


Student Instructor 
Ratio 




1.18 


1.16 


1 


.55 


Aircraft Flights 


First Tour 


8 




5 




per Student 


Second Tour 


6 




5 




Simulator Periods 


First Tour 


0 




7 




per Student 


Second Tour 


0 




6 




Allotted Ground 


First Tour 


196 




232 




School Time (Hrs) 


Second Tour 


162 




207 




Syllabus Length 


First Tour 


9 wks 
3 days 




10 


wks 




Second Tour 


8 wks 




6 


wks 






2 days 




6 


days 
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As illustrated in Table 13, the student/instructor ratio was 
increased between the pre-ISD period of FY77 and FY78 and the 
ISD period of FY79 by approximately .39. This increase 
occurred concurrently with an expansion of the second-tour 
pilot load and would indicate that the efficiency of self- 
instruction associated with ISD was being realized. Ground 
school time specified for the familiarization phase was 
increased between FY77 and FY79 due to the introduction of 
the flight simulator and the inclusion of simulator time in 
ground school totals. In aggregate, operational data seem 
to have remained essentially constant over the time frame 
examined, except for a modest, but steady, increase in the 
enlisted staff size. 
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IV. DISCUSSION 



The preceding analysis suggests that VP-31 pilot training 
under the new ISD condition serves to account only for lower 
scores on force-level, closed-book, NATOPS examinations by 
pilots assigned to one, of two, west-coast P-3 bases. Further, 
this study identified rank, squadron experience, and squadron 
location as statistically significant effects in accounting 

for the variance of examination scores. These results raise 

\ 

three salient questions regarding P-3 pilot training and sub- 
sequent NATOPS evaluation. 

1. Is the pilot training provided by VP-31 in conformance 
with the best and proper application of ISD? 

2. Is the force-level NATOPS testing program an ap- 
propriate measure of the effectiveness of VP-31 
pilot training? 

3. Is ISD a more effective system for teaching pilots 
to learn, retain, and apply a knowledge of P-3 
operating procedures and systems than the previous 
traditional system? 

To determine the degree of conformance between current 
training at VP-31 and the best and proper application of ISD 
would require a formative evaluation, the magnitude of which 
is beyond the scope of this study. On the surface however, 
it appears that with but a few exceptions, such as the lack 
of entry level testing and a true self-paced learning program 
which would allow students to complete training at a self 
determined schedule, the basic principles of ISD are being 
followed. Courseware Incorporated, a private-sector ISD 
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consultant, has been largely responsible for the design of 
VP-31 's new training program. 

The appropriateness of the force-level NATOPS testing 
program as a measure of VP-31 pilot training effectiveness 
was established by the close agreement between course objec- 
tives and the areas specified for evaluation in the NATOPS 
program. (See Appendix A for a detailed listing.) Although 
a certain degree of latitude is permitted in the construction 
of individual examination questions, the specific subject 
matter is clearly defined for both open- and closed-book 
examinations. The open-book examination is a particularly 
appropriate measure of training effectiveness because in 
general it requires pilots to compute values associated with 
aircraft performance which are identical to those required on 
the job. Analysis of the variance of open-book examination 
scores suggests that among the variables considered, the 
effects of experience (rank and months) and squadron location 
are statistically significant. Further, the effect of the 
training variable (ISD or non-ISD) was not significant. 

Closed-book examinations measure primarily verbal infor- 
mation and intellectual skills associated with operating 
procedures and aircraft systems. Since the examination re- 
quires pilots to first read and interpret a question before 
selecting a written answer, the relationship to on-the-job 
requirements is not as close as in the open-book examination. 
Analysis of the closed-book examination scores revealed, as 
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with open-book scores, a sensitivity to the rank and months 
variables. The type of training received was significant at 
the .08 level as a main effect and at the .03 level when 
interacted with the squadron location variable (ISD trained 
pilots scored lower than traditionally trained pilots at 
location A) . The direction and consistency of the main effect 
is very obvious in Table 4. Of the 14 contrasts, in all but 
one the ISD-trained pilots have lower closed-book examination 
scores . 

The preceding discussion suggests that organizational 
climate, as v;ell as experience, is strongly related to pilot 
performance on NATOPS examinations. Formal syllabus training 
received at VP-31, either ISD or traditional, serves to 
provide pilots with a basic knowledge of P-3 systems and 
procedures. Retention of these skills and further learning 
takes place in the field at the squadron level where previous- 
ly learned skills have the opportunity to grow or deteriorate. 
Under a positive climate, learning is reinforced and perfor- 
mance is rewarded. Measurement of the learning climate 
associated with an organization should be based upon the 
attitudes of individuals which result from choices, such as 
whether or not to maintain previous learning and further 
develop aviator skills. 

The ISD method focuses sharply upon the development of a 
precise list of learning objectives based upon task analyses. 
Although the logic of the atomized approach toward learning 
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specified in design manuals is sound, course designers must 
exercise particular attention to detail when assembling task 
inventories and subsequent learning objectives. If graduates 
are deficient in an area not listed among the objectives, the 
problem is with the list and not with the program. The 
learning of attitudes, which is not emphasized within the 
Navy's ISD design manual, should be considered by course 
designers as a desired outcome of pilot training. 

This study seems to indicate that perhaps a truly positive 
attitude toward the further development of knowledge in the 
squadron environment is not being learned as well under the 
ISD system as it was under the traditional methods. Something 
may have been overlooked in the time-consuming atomization 
process of ISD when learning became limited to those items 
on the list. Additionally, emphasis on the individualization 
of instruction and self-instruction may have served to 
eliminate the positive effects of peer and instructor inter- 
actions in the learning of favorable attitudes. 
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V. CONCLUSION 



This study suggests that by utilizing force- level NATOPS 
examination scores as criteria for the effectiveness of pilot 
training conducted at VP-31, enhancement by the application 
of ISD has not yet been achieved. The following findings are 
germane; 

1. A reduction in closed-book NATOPS examination scores 
associated with ISD trained pilots, particularly at one of 
two west-coast P-3 bases, was identified. Although these ISD 
trained pilots did score lower on average than their tradition- 
ally trained counterparts, it is somewhat presumptuous to 
attribute the difference to the process of ISD until a forma- 
tive evaluation is conducted at VP-31 to establish the degree 
to which the pilot-training program conforms to the best and 
proper utilization of ISD. 

2. Upon completion of a formative evaluation, further 
summative evaluation, under the controlled conditions of 
parallel training previously discussed, is required to 
determine the effectiveness of a program verified to be a 
proper application of ISD. An evaluation of this type is 
necessary to justify the extensive cost in terms of manpower 
and money allocated to the development of ISD at VP-31. 

3. Pilot training provided by VP-31 during the time 
frame examined (either ISD or non-ISD) seems to have a 
limited effect upon force-level NATOPS examination scores. 
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However, the effects of squadron location and experience (rank 
and months) are statistically significant in accounting for 
the variance of NATOPS scores. 
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APPENDIX A 



A COMPARISON OF TRAINING OBJECTIVES AND THE SUBJECT 
MATTER OF NATOPS EXAMINATIONS 



Examination Subject Training Objective Unit 



Open Book Examination 



Ordnance/Zero Fuel 
Weight Computation 

Fuel Density /Minimum 
Fuel for Flight 

Center of Gravity 
Limits 



Define and Compute 
Zero Fuel Weight 

Utilize Minimum Fuel 
Flight Chart 

Define and Compute 
Center of Gravity, 
State Limits 



1 

1 

1 



Minimum Control Speed 
Runway Wind Components 

Refusal Speed 
VR, RCR 

Vro, Vlof, V50 (3) 
Distance to Liftoff 



Compute VMC 

Compute Runway Wind 
Component 

Compute Refusal Speed 

Compute Correction to 
Refusal Speed for RCR 

Compute Take Off Speeds 
Vro, Vlof, V50 (3&4 Eng) 

Compute distance to 
Liftoff 



3 

3 

3 

3 

3 

3 



Stall Speed/Stall Compute Stall Speed 3 

Buffet 



Climb Performance Compute 3/4 Eng Climb- 3 

out Flight Path 

Altitude Selection Compute Max Range 3 

Altitude 



Max Range Airspeed 



Max Range SHP 



Compute Max Range Air- 3 

speed 

Compute Max Range SHP 3 



Lesson # 



8 

8 

8 



1 

1 

1 

1 

1 

1 

1 

2 

4 

6 

6 
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Unit Lesson # 



Examination Subject 



Training Objective 



Max Range Fuel 
Planning 

Loiter Time Predic- 
tion 



Utilize Max Range 
Charts 

Compute Loiter Time 
Available 



3 

3 



4 

5 



Angle of Attack 



Define AOA 



1 25 



Composite Cruise 
Flight Performance 
Chart 



Utilize Composite 
Cruise Performance 
Chart 



3 



7 



Ditch Speed 



Compute Ditch Speed 4 12 



Available Range Compute Available Range 3 8 

Remaining Remaining 



Landing Ground Roll 
Distance 

NATOPS Manual/ 
Evaluations 

Emergencies 



Compute Landing Ground 3 3 

Roll 

NATOPS Program Review 1 2 

Emergencies covered in 4 all 



Closed Book Examination 



Aircraft General 



General Aircraft 



1 



Power Plants 



Propeller 



Electrical System 



Hydraulic System 



Auxiliary Power Unit 



Describe the Operation 1 

and Limits of the 
Engines 

Describe the Operation 1 

and Limits of the 
Propeller 

Describe the Operation 1 

and Limits of the 
Electrical System 

Describe the Operation 1 

and Limits of the 
Hydraulic System 

Describe the Operation 1 

and Limits of the APU 



9 



15,16, 

17 

3, 4, 

5 

6 
13 
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Unit Lesson # 



Examination Subject 
Flight Controls 

Automatic Flight 
Control System 

Air Conditioning and 
Pressurization 

Fuel System 

Radio and Nav Aids 

All Weather Systems 

Armament 

Instruments 

Weight and Balance 

Aircraft Performance 
and Cruise Control 

Ditching and Bailout 

Normal and Emergency 
Operating Procedures 

Limitations 



Training Objective 



Describe the Operation 
of the Flight Controls 


1 


6 


Describe the Operation 
and Limits of the 
Automatic Flight Control 
System 


1 


26, 


Describe the Operation 
and Limits of the Air 
Conditioning and 
Pressurization System 


1 


18, 


Describe the Operation 
and Limits of the Fuel 
System 


1 


11, 


Describe the Operation 
of the Radios and Nav 
Aids 


1 


21, 

23, 

25 


Describe the Operation 1 

and Limits of the Foul 
Weather Systems 

Covered in Tactics Section 


20 


Describe the function 
and limits of cockpit 
instruments 


1 


25 


Define and Compute Weight 
and Balance Data 


1 


8 


Compute Max Range Cruise 


3 


6 


Describe Procedures for 
Ditch and Bailout 


4 


12 


Normal and Emergency 
Operating Procedures 


2&4 


all 


Normal and Emergency 
Operating Procedures 


2&4 


all 
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